Video distribution system for live distributing video containing animation of character object generated based on motion of actors

ABSTRACT

A video distribution system in one embodiment includes a distribution server, a storage and a supporter computer. The distribution server may be configured to cause a first display device and a plurality of client devices to play a live video containing an animation of a character object. A supporter computer may be configured to display in the live video, upon selection of a decorative object and receipt of a first operation input, a blind object for hiding at least a part of the character object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation Application of U.S. Ser. No.16/407,733 (filed on May 9, 2019), which claims the benefit of priorityfrom Japanese Patent Application Serial No. 2018-90907 (filed on May 9,2018), and Japanese Patent Application Serial No. 2018-224331 (filed onNov. 30, 2018), the contents of each of which are hereby incorporated byreference in its entirety.

TECHNICAL FIELD

The present disclosure relates to a video distribution system for livedistribution of a video containing animation of a character objectgenerated based on motions of an actor.

BACKGROUND

Video distribution systems that produce an animation of a characterobject based on actor's motions and live distribute a video containingthe animation of the character object have been known. Such a videodistribution system is disclosed, for example, in Japanese PatentApplication Publication No. 2015-184689 (“the '689 Publication”).

During a live distribution of a video, unexpected situations such as afailure of a device that is used to generate animation may occur. In theabove-mentioned conventional video distribution system, the livedistribution of the video is carried out only by an actor and anoperator of a camera. If the actor or the camera operator has to handlesuch an unexpected situation during the live distribution, his/hermotions to work on the failure or the like may be reflected in motionsof a character object, or inappropriate camerawork may occur.Consequently, the quality of the live distributed video may be adverselyaffected.

SUMMARY

It is an object of the present disclosure to provide a technicalimprovement which solves or alleviates at least part of the drawbacks ofthe prior art mentioned above. More specifically, one object of thedisclosure is to provide a video distribution system that makes iteasier to handle with an unexpected situation occurred during a livedistribute.

A video distribution system according to one aspect includes adistribution server configured to live distribute a video that containsan animation of a character object generated based on actor's motions toa first client device used by a first user, a first display devicedisposed at a position viewable by the actor and displaying the video,and a supporter computer causing first additional information to bedisplayed in the video displayed on the first display device based on afirst operation input while causing the first additional information tobe undisplayed on the first client device.

In the video distribution system, the supporter computer may causesecond additional information to be displayed in the video displayed onthe first display device and in the video displayed on the first clientdevice based on a second operation input.

In the video distribution system, the supporter computer may cause amodified character object in which at least a part of the characterobject is modified to be generated based on a third operation input andcause the modified character object to be included in the video.

In the video distribution system, the character object may have a faceportion that is generated so as to move based on face motion datarepresenting face motions of the actor, and the distribution serverdevice may display third additional information in the face portionbased on the third operation input.

The video distribution system may further include a storage storing adecorative object displayed in association with the character object,and the supporter computer may display a blind object for hiding atleast a part of the character object when the decorative object isdisplayed in the video based on a forth operation input.

In the video distribution system, the distribution server device may beconfigured to live distribute the video to a second client device usedby a second user in addition to the first client device, the firstclient device is a first type device, the second client device is asecond type device different from the first type device, and thesupporter computer may include a second display device that displays adisplay image of the video displayed on the first client device and adisplay image of the video displayed on the second client device.

In the video distribution system, the supporter computer may prohibitdistribution of the video to the first user based on a fifth operationinput.

Another aspect of the invention relates to a method of distributing avideo performed by executing computer readable instructions by one ormore computer processor. The method includes live distributing a videothat contains an animation of a character object generated based onactor's motions to a first client device used by a first user,displaying the video on a first display device disposed at a positionviewable by the actor, and displaying first additional information inthe video displayed on the first display device based on a firstoperation input while causing the first additional information to beundisplayed on the first client device.

Yet another aspect of the invention relates to a computer-readabletangible non-transitory storage medium including a program executed byone or more computer processors. The computer program causes the one ormore computer processors to perform live distributing a video thatcontains an animation of a character object generated based on actor'smotions to a first client device used by a first user, displaying thevideo on a first display device disposed at a position viewable by theactor, and displaying first additional information in the videodisplayed on the first display device based on a first operation inputwhile causing the first additional information to be undisplayed on thefirst client device.

According to the aspects mentioned above, it is possible to provide avideo distribution system that makes it easier to handle with anunexpected situation occurred during a live distribution.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a video distribution systemaccording to one embodiment.

FIG. 2 schematically illustrates a installation of a studio whereproduction of a video that is distributed in the video distributionsystem of FIG. 1 is performed.

FIG. 3 illustrates a possession list stored in the video distributionsystem of FIG. 1 .

FIG. 4 illustrates a candidate list stored in the video distributionsystem of FIG. 1 .

FIG. 5 illustrates an example of a video displayed on the client device10 a in one embodiment. An animation of a character object is includedin FIG. 5 .

FIG. 6 illustrates an example of a video displayed on the client device10 a in one embodiment. A normal object is included in FIG. 6 .

FIG. 7 illustrates an example of a video displayed on the client device10 a in one embodiment. A decorative object is included in FIG. 7 .

FIG. 8 schematically illustrates an example of a decorative objectselection screen for selecting a desired decorative object from amongthe decorative objects included in the candidate list.

FIG. 9 illustrates an example of an image displayed on a display 44 of asupporter computer 40 in one embodiment.

FIG. 10 illustrates an example of an image displayed on a display 39 ofa supporter computer 40 and an example of first additional informationin one embodiment.

FIG. 11 a illustrates an example of a video displayed on the clientdevice 10 a in one embodiment, and FIG. 11 a includes second additionalinformation.

FIG. 11 b illustrates an example of the video displayed on the clientdevice 10 a in one embodiment, and FIG. 11 b includes the secondadditional information.

FIG. 12 a illustrates an example of a video displayed on the clientdevice 10 a in one embodiment, and FIG. 12 a includes third additionalinformation.

FIG. 12 b illustrates an example of the video displayed on the clientdevice 10 a in one embodiment, and FIG. 12 b includes third additionalinformation.

FIG. 13 illustrates an example of a video displayed on the client device10 a in one embodiment, and FIG. 13 includes a blind object.

FIG. 14 is a flow chart showing a flow of a video distribution processin one embodiment.

FIG. 15 is a flowchart of a process of displaying the first additionalinformation in one embodiment.

FIG. 16 is a flowchart of a process of displaying the second additionalinformation in one embodiment.

FIG. 17 is a flowchart of a process of displaying the third additionalinformation in one embodiment.

FIG. 18 is a flowchart of a process of displaying a blind object in oneembodiment.

FIG. 19 is a flow chart showing a flow of a process for prohibitingvideo distribution to a banned user in one embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Various embodiments of the disclosure will be described hereinafter withreference to the accompanying drawings. Throughout the drawings, thesame or similar elements are denoted by the same reference numerals.

With reference to FIGS. 1 to 4 , a video distribution system accordingto an embodiment will be described. FIG. 1 is a block diagramillustrating a video distribution system 1 according to one embodiment,FIG. 2 schematically illustrates an installation of a studio where avideo to be distributed in the video distribution system 1 is produced,and FIGS. 3 to 4 are for describing information stored in the videodistribution system 1.

The video distribution system 1 includes client devices 10 a to 10 c, aserver device 20, a studio unit 30, and a storage 60. The client devices10 a to 10 c, the server device 20, and the storage 60 are communicablyinterconnected over a network 50. The server device 20 is configured todistribute a video including an animation of a character, as describedlater. The character included in the video may be motion controlled in avirtual space.

The video may be distributed from the server device 20 to each of theclient devices 10 a to 10 c. A first viewing user who is a user of theclient device 10 a, a second viewing user who is a user of the clientdevice 10 b, and a third viewing user who is a user of the client device10 c are able to view the distributed video with their client device.The video distribution system 1 may include less than three clientdevices, or may include more than three client devices. In thisspecification, when it is not necessary to distinguish the clientdevices 10 a to 10 c and other client devices from each other, they maybe collectively referred to as “client devices.”

The client devices 10 a to 10 c are information processing devices suchas smartphones. In addition to the smartphone, the client devices 10 ato 10 c each may be a mobile phone, a tablet, a personal computer, anelectronic book reader, a wearable computer, a game console, or anyother information processing devices that are capable of playing videos.Each of the client devices 10 a to 10 c may include a computerprocessor, a memory unit, a communication I/F, a display, a sensor unitincluding various sensors such as a gyro sensor, a sound collectingdevice such as a microphone, and a storage for storing variousinformation.

Viewing users are able to input a message regarding the distributedvideo via an input interface of the client devices 10 a to 10 c to postthe message to the server device 20. The message posted from eachviewing user may be displayed such that it is superimposed on the video.In this way, interaction between viewing users is achieved.

In the illustrated embodiment, the server device 20 includes a computerprocessor 21, a communication I/F 22, and a storage 23.

The computer processor 21 is a computing device which loads variousprograms realizing an operating system and various functions from thestorage 23 or other storage into a memory unit and executes instructionsincluded in the loaded programs. The computer processor 21 is, forexample, a CPU, an MPU, a DSP, a GPU, any other computing device, or acombination thereof. The computer processor 21 may be realized by meansof an integrated circuit such as ASIC, PLD, FPGA, MCU, or the like.Although the computer processor 21 is illustrated as a single componentin FIG. 1 , the computer processor 21 may be a collection of a pluralityof physically separate computer processors. In this specification, aprogram or instructions included in the program that are described asbeing executed by the computer processor 21 may be executed by a singlecomputer processor or distributed and executed by a plurality ofcomputer processors. Further, a program or instructions included in theprogram executed by the computer processor 21 may be executed by aplurality of virtual computer processors.

The communication I/F 22 may be implemented as hardware, firmware, orcommunication software such as a TCP/IP driver or a PPP driver, or acombination thereof. The server device 20 is able to transmit andreceive data to and from other devices via the communication I/F 22.

The storage 23 is an storage device accessed by the computer processor21. The storage 23 is, for example, a magnetic disk, an optical disk, asemiconductor memory, or various other storage device capable of storingdata. Various programs may be stored in the storage 23. At least some ofthe programs and various data that may be stored in the storage 23 maybe stored in a storage (for example, a storage 60) that is physicallyseparated from the server device 20.

Most of components of the studio unit 30 are disposed, for example, in astudio room R shown in FIG. 2 . As illustrated in FIG. 2 , an actor A1and an actor A2 give performances in the studio room R. The studio unit30 is configured to detect motions and expressions of the actor A1 andthe actor A2, and to output the detection result information to theserver device 20.

Both the actor A1 and the actor A2 are objects whose motions andexpressions are captured by a sensor group provided in the studio unit30, which will be described later. The actor A1 and the actor A2 are,for example, humans, animals, or moving objects that give performances.The actor A1 and the actor A2 may be, for example, autonomous robots.The number of actors in the studio room R may be one or three or more.

The studio unit 30 includes six motion sensors 31 a to 31 f attached tothe actor A1, a controller 33 a held by the left hand of the actor A1, acontroller 33 b held by the right hand of the actor A1, and a camera 37a attached to the head of the actor A1 via an attachment 37 b. Thestudio unit 30 also includes six motion sensors 32 a to 32 f attached tothe actor A2, a controller 34 a held by the left hand of the actor A2, acontroller 34 b held by the right hand of the actor A2, and a camera 38a attached to the head of the actor A2 via an attachment 38 b. Amicrophone for collecting audio data may be provided to each of theattachment 37 b and the attachment 38 b. The microphone can collectspeeches of the actor A1 and the actor A2 as voice data. The microphonesmay be wearable microphones attached to the actor A1 and the actor A2via the attachment 37 b and the attachment 38 b. Alternatively themicrophones may be installed on the floor, wall or ceiling of the studioroom R. In addition to the components described above, the studio unit30 includes a base station 35 a, a base station 35 b, a tracking sensor36 a, a tracking sensor 36 b, and a display 39. A supporter computer 40is installed in a room next to the studio room R, and these two roomsare separated from each other by a glass window. The server device 20may be installed in the same room as the room in which the supportercomputer 40 is installed.

The motion sensors 31 a to 31 f and the motion sensors 32 a to 32 fcooperate with the base station 35 a and the base station 35 b to detecttheir position and orientation. In one embodiment, the base station 35 aand the base station 35 b are multi-axis laser emitters. The basestation 35 a emits flashing light for synchronization and then emits alaser beam about, for example, a vertical axis for scanning. The basestation 35 a emits a laser beam about, for example, a horizontal axisfor scanning. Each of the motion sensors 31 a to 31 f and the motionsensors 32 a to 32 f may be provided with a plurality of optical sensorsfor detecting incidence of the flashing lights and the laser beams fromthe base station 35 a and the base station 35 b, respectively. Themotion sensors 31 a to 31 f and the motion sensors 32 a to 32 f each maydetect a time difference between an incident timing of the flashinglight and an incident timing of the laser beam, time when each opticalsensor receives the light and or beam, an incident angle of the laserlight detected by each optical sensor, and any other information asnecessary. The motion sensors 31 a to 31 f and the motion sensors 32 ato 32 f may be, for example, Vive Trackers provided by HTC CORPORATION.The base station 35 a and the base station 35 b may be, for example,base stations provided by HTC CORPORATION. Three or more base stationsmay be provided. The position of the base station may be changed asappropriate. For example, in addition to or instead of the base stationdisposed at the upper corner of the space to be detected by the trackingsensor, a pair of the base stations may be disposed at a upper positionand a lower position close to the floor.

Detection information (hereinafter may also be referred to as “trackinginformation”) obtained as a result of detection performed by each of themotion sensors 31 a to 31 f and the motion sensors 32 a to 32 f istransmitted to the server device 20. The detection result informationmay be wirelessly transmitted to the server device 20 from each of themotion sensors 31 a to 31 f and the motion sensors 32 a to 32 f. Sincethe base station 35 a and the base station 35 b emit flashing light anda laser light for scanning at regular intervals, the trackinginformation of each motion sensor is updated at each interval. Theposition and the orientation of each of the motion sensors 31 a to 31 fand the motion sensors 32 a to 32 f may be calculated based on thetracking information detected by the motion sensors 31 a to 31 f and themotion sensors 32 a to 32 f. The position and the orientation of each ofthe motion sensors 31 a to 31 f and the motion sensors 32 a to 32 f maybe calculated based on, for example, the tracking information in theserver device 20.

In the illustrated embodiment, the six motion sensors 31 a to 31 f aremounted on the actor A. The motion sensors 31 a, 31 b, 31 c, 31 d, 31 e,and 31 f are attached to the left wrist, the right wrist, the leftinstep, the right instep, the hip, and top of the head of the actor A1,respectively. The motion sensors 31 a to 31 f may each be attached tothe actor A1 via an attachment. The six motion sensors 32 a to 32 f aremounted on the actor A2. The motion sensors 32 a to 32 f may be attachedto the actor A2 at the same positions as the motion sensors 31 a to 31f. The motion sensors 31 a to 31 f and the motion sensors 32 a to 32 fshown in FIG. 2 are merely an example. The motion sensors 31 a to 31 fmay be attached to various parts of the body of the actor A1, and themotion sensors 32 a to 32 f may be attached to various parts of the bodyof the actor A2. The number of motion sensors attached to the actor A1and the actor A2 may be less than or more than six. As described above,body movements of the actor A1 and the actor A2 are detected bydetecting the position and the orientation of the motion sensors 31 a to31 f and the motion sensors 32 a to 32 f attached to the body parts ofthe actor A1 and the actor A2.

In one embodiment, a plurality of infrared LEDs are mounted on each ofthe motion sensors attached to the actor A1 and the actor A2, and lightfrom the infrared LEDs are sensed by infrared cameras provided on thefloor and/or wall of the studio room R to detect the position and theorientation of each of the motion sensors. Visible light LEDs may beused instead of the infrared LEDs, and in this case light from thevisible light LEDs may be sensed by visible light cameras to detect theposition and the orientation of each of the motion sensors. As describedabove, a light emitting unit (for example, the infrared LED or visiblelight LED) may be provided in each of the plurality of motion sensorsattached to the actor, and a light receiving unit (for example, theinfrared camera or visible light camera) provided in the studio room Rsenses the light from the light emitting unit to detect the position andthe orientation of each of the motion sensors.

In one embodiment, a plurality of reflective markers may be used insteadof the motion sensors 31 a-31 f and the motion sensors 32 a-32 f. Thereflective markers may be attached to the actor A1 and the actor A2using an adhesive tape or the like. The position and orientation of eachreflective marker can be estimated by capturing images of the actor A1and the actor A2 to which the reflective markers are attached togenerate captured image data and performing image processing on thecaptured image data.

The controller 33 a and the controller 33 b supply, to the server device20, control signals that correspond to operation of the actor A1.Similarly, the controller 34 a and the controller 34 b supply, to theserver device 20, control signals that correspond to operation of theactor A2.

The tracking sensor 36 a and the tracking sensor 36 b generate trackinginformation for determining configuration information of a virtualcamera used for constructing a virtual space included in the video. Thetracking information of the tracking sensor 36 a and the tracking sensor36 b is calculated as the position in its three-dimensional orthogonalcoordinate system and the angle around each axis. The position andorientation of the tracking sensor 36 a may be changed according tooperation of the operator. The tracking sensor 36 a transmits thetracking information indicating the position and the orientation of thetracking sensor 36 a to the tracking information server device 20.Similarly, the position and the orientation of the tracking sensor 36 bmay be set according to operation of the operator. The tracking sensor36 b transmits the tracking information indicating the position and theorientation of the tracking sensor 36 b to the tracking informationserver device 20. The tracking sensors 36 a, 36 b each may be supportedby a gimbal or stabilizer. The gimbal may be configured to be graspableby an actor or a supporter. By supporting the tracking sensors 36 a and36 b with the gimbals or stabilizers, it is possible to suppress camerashake.

The camera 37 a is attached to the head of the actor A1 as describedabove. For example, the camera 37 a is disposed so as to capture animage of the face of the actor A1. The camera 37 a continuously capturesimages of the face of the actor A1 to obtain imaging data of the face ofthe actor A1. Similarly, the camera 38 a is attached to the head of theactor A2. The camera 38 a is disposed so as to capture an image of theface of the actor A2 and continuously capture images of the face of theactor A2 to obtain captured image data of the face of the actor A2. Thecamera 37 a transmits the captured image data of the face of the actorA1 to the server device 20, and the camera 38 a transmits the capturedimage data of the face of the actor A2 to the server device 20. Thecamera 37 a and the camera 38 a may be 3D cameras capable of detectingthe depth of a face of a person.

The display 39 is configured to display information received from thesupporter computer 40. The information transmitted from the supportercomputer 40 to the display 39 may include, for example, textinformation, image information, and various other information. Thedisplay 39 is disposed at a position where the actor A1 and the actor A2are able to see the display 39. The display 39 is an example of a firstdisplay device.

In the illustrated embodiment, the supporter computer 40 is installed inthe next room of the studio room R. Since the room in which thesupporter computer 40 is installed and the studio room R are separatedby the glass window, an operator of the supporter computer 40 (sometimesreferred to as “supporter” in the specification) is able to see theactor A1 and the actor A2. In the illustrated embodiment, supporters B1and B2 are present in the room as the operators of the supportercomputer 40. In the specification, the supporter B1 and the supporter B2may be collectively referred to as the “supporter” when it is notnecessary to distinguish them from each other.

The supporter computer 40 may be configured to be capable of changingthe setting(s) of the component(s) of the studio unit 30 according tothe operation by the supporter B1 and the supporter B2. The supportercomputer 40 can change, for example, the setting of the scanninginterval performed by the base station 35 a and the base station 35 b,the position or orientation of the tracking sensor 36 a and the trackingsensor 36 b, and various settings of other devices. At least one of thesupporter B1 and the supporter B2 is able to input a message to thesupporter computer 40, and the input message is displayed on the display39.

The components and functions of the studio unit 30 shown in FIG. 2 aremerely example. The studio unit 30 applicable to the invention mayinclude various constituent elements that are not shown. For example,the studio unit 30 may include a projector. The projector is able toproject a video distributed to the client device 10 a or another clientdevice on the screen S.

Next, information stored in the storage 23 in one embodiment will bedescribed. In the illustrated embodiment, the storage 23 stores modeldata 23 a, object data 23 b, a possession list 23 c, a candidate list 23d, and any other information required for generation and distribution ofa video to be distributed.

The model data 23 a is model data for generating animation of acharacter. The model data 23 a may be three-dimensional model data forgenerating three-dimensional animation, or may be two-dimensional modeldata for generating two-dimensional animation. The model data 23 aincludes, for example, rig data (also referred to as “skeleton data”)indicating a skeleton of a character, and surface data indicating theshape or texture of a surface of the character. The model data 23 a mayinclude two or more different pieces of model data. Pieces of model datamay each have different rig data, or may have the same rig data. Thepieces of model data may have surface data different from each other ormay have the same surface data. In the illustrated embodiment, in orderto generate a character object corresponding to the actor A1 and acharacter object corresponding to the actor A2, the model data 23 aincludes at least two types of model data different from each other. Themodel data for the character object corresponding to the actor A1 andthe model data for the character object corresponding to the actor A2may have, for example, the same rig data but different surface data fromeach other.

The object data 23 b includes asset data used for constructing a virtualspace in the video. The object data 23 b includes data for rendering abackground of the virtual space in the video, data for rendering variousobjects displayed in the video, and data for rendering any other objectsdisplayed in the video. The object data 23 a may include object positioninformation indicating the position of an object in the virtual space.

In addition to the above, the object data 23 b may include a gift objectdisplayed in the video in response to a display request from viewingusers of the client devices 10 a to 10 c. The gift object may include aneffect object, a normal object, and a decorative object. Viewing usersare able to purchase a desired gift object. Moreover, an upper limit maybe set for the number of objects that a viewing user is allowed topurchase or the amount of money that the viewing user is allowed tospend for objects.

The effect object is an object that affects the impression of the entireviewing screen of the distributed video, and is, for example, an objectrepresenting confetti. The object representing confetti may be displayedon the entire viewing screen, which can change the impression of theentire viewing screen before and after its display. The effect objectmay be displayed so as to overlap with the character object, but it isdifferent from the decorative object in that it is not displayed inassociation with a specific portion of the character object.

The normal object is an object functioning as a digital gift from aviewing user (for example, the actor A1 or the actor A2) to an actor,for example, an object resembling a stuffed toy or a bouquet. In oneembodiment, the normal object is displayed on the display screen of thevideo such that it does not contact the character object. In oneembodiment, the normal object is displayed on the display screen of thevideo such that it does not overlap with the character object. Thenormal object may be displayed in the virtual space such that itoverlaps with an object other than the character object. The normalobject may be displayed so as to overlap with the character object, butit is different from the decorative object in that it is not displayedin association with a specific portion of the character object. In oneembodiment, when the normal object is displayed such that it overlapswith the character object, the normal object may hide portions of thecharacter object other than the head including the face of the characterobject but does not hide the head of the character object.

The decorative object is an object displayed on the display screen inassociation with a specific part of the character object. In oneembodiment, the decorative object displayed on the display screen inassociation with a specific part of the character object is displayedadjacent to the specific part of the character object on the displayscreen. In one embodiment, the decorative object displayed on thedisplay screen in association with a specific part of the characterobject is displayed such that it partially or entirely covers thespecific part of the character object on the display screen.

The decorative object is an object that can be attached to a characterobject, for example, an accessory (such as a headband, a necklace, anearring, etc.), clothes (such as a T-shirt), a costume, and any otherobject which can be attached to the character object. The object data 23b corresponding to the decorative object may include attachment positioninformation indicating which part of the character object the decorativeobject is associated with. The attachment position information of adecorative object may indicate to which part of the character object thedecorative object is attached. For example, when the decorative objectis a headband, the attachment position information of the decorativeobject may indicate that the decorative object is attached to the “head”of the character object. When the decorative object is a T-shirt, theattachment position information of the decorative object may indicatethat the decorative object is attached to the “torso” of the characterobject.

A duration of time of displaying the gift objects may be set for eachgift object depending on its type. In one embodiment, the duration ofdisplaying the decorative object may be set longer than the duration ofdisplaying the effect object and the duration of displaying the normalobject. For example, the duration of displaying the decorative objectmay be set to 60 seconds, while the duration of displaying the effectobject may be set to five seconds and the duration of displaying thenormal object may be set to ten seconds.

The possession list 23 c is a list showing gift objects possessed byviewing users of a video. An example of the possession list 23 c isshown in FIG. 3 . As illustrated, in the possession list 23 c, an objectID for identifying a gift object possessed by a viewing user is storedin association with account information of the viewing user (forexample, user ID of the viewing user). The viewing users include, forexample, the first to third viewing users of the client devices 10 a to10 c.

The candidate list 23 d is a list of decorative objects for which adisplay request has been made from a viewing user. As will be describedlater, a viewing user who holds a decorative object(s) is able to make arequest to display his/her possessed decorative objects. In thecandidate list 23 d, object IDs for identifying decorative objects arestored in association with the account information of the viewing userwho has made a request to display the decorative objects. The candidatelist 23 d may be created for each distributor. The candidate list 23 dmay be stored, for example, in association with distributoridentification information that identify a distributor(s) (the actor A1,the actor A2, the supporter B1, and/or the supporter B2).

Functions realized by the computer processor 21 will be now describedmore specifically. The computer processor 21 functions as a body motiondata generation unit 21 a, a face motion data generation unit 21 b, ananimation generation unit 21 c, a video generation unit 21 d, a videodistribution unit 21 e, a display request processing unit 21 f, adecorative object selection unit 21 g, and an object purchase processingunit 21 h by executing computer-readable instructions included in adistributed program. At least some of the functions that can be realizedby the computer processor 21 may be realized by a computer processorother than the computer processor 21 of the video distribution system 1.For example, at least some of the functions realized by the computerprocessor 21 may be realized by a computer processor mounted on thesupporter computer 40.

The body motion data generation unit 21 a generates first body motiondata of each part of the body of the actor A1 based on the trackinginformation obtained through detection performed by the correspondingmotion sensors 31 a to 31 f, and generates second body motion data,which is a digital representation of the position and the orientation ofeach part of the body of the actor A2, based on the tracking informationobtained through detection performed by the corresponding motion sensors32 a to 32 f. In the specification, the first body motion data and thesecond body motion data may be collectively referred to simply as “bodymotion data.” The body motion data is serially generated with time asneeded. For example, the body motion data may be generated atpredetermined sampling time intervals. Thus, the body motion data canrepresent body movements of the actor A1 and the actor A2 in time seriesas digital data. In the illustrated embodiment, the motion sensors 31 ato 31 f and the motion sensors 32 a to 32 f are attached to the left andright limbs, the waist, and the head of the actor A1 and the actor A2,respectively. Based on the tracking information obtained throughdetection performed by the motion sensors 31 a to 31 f and the motionsensors 32 a to 32 f, it is possible to digitally represent the positionand orientation of the substantially whole body of the actor A1 and theactor A2 in time series. The body motion data can define, for example,the position and rotation angle of bones corresponding to the rig dataincluded in the model data 23 a.

The face motion data generation unit 21 b generates first face motiondata, which is a digital representation of motions of the face of theactor A1, based on captured image data of the camera 37 a, and generatessecond face motion data, which is a digital representation of motions ofthe face of the actor A2, based on captured image data of the camera 38a. In the specification, the first face motion data and the second facemotion data may be collectively referred to simply as “face motiondata.” The face motion data is serially generated with time as needed.For example, the face motion data may be generated at predeterminedsampling time intervals. Thus, the face motion data can digitallyrepresent facial motions (changes in facial expression) of the actor A1and the actor A2 in time series.

The animation generation unit 21 c is configured to apply the bodymotion data generated by the body motion data generation unit 21 a andthe face motion data generated by the face motion data generation unit21 b to predetermined model data included in the model data 23 a inorder to generate an animation of a character object that moves in avirtual space and whose facial expression changes. More specifically,the animation generation unit 21 c may generate an animation of acharacter object moving in synchronization with the motions of the bodyand facial expression of the actor A1 based on the first body motiondata and the first face motion data related to the actor A1, andgenerate an animation of a character object moving in synchronizationwith the motions of the body and facial expression of the actor A2 basedon the second body motion data and the second face motion data relatedto the actor A2. In the specification, a character object generatedbased on the motion and expression of the actor A1 may be referred to asa “first character object”, and a character object generated based onthe motion and expression of the actor A2 may be referred to as a“second character object.”

The video generation unit 21 d constructs a virtual space using theobject data 23 b, and generates a video that includes the virtual space,the animation of the first character object corresponding to the actorA1, and the animation of the second character object corresponding tothe actor A2. The first character object is disposed in the virtualspace so as to correspond to the position of the actor A1 with respectto the tracking sensor 36 a, and the second character object is disposedin the virtual space so as to corresponds to the position of the actorA2 with respect to the tracking sensor 36 a. Thus, it is possible tochange the position and the orientation of the first character objectand the second character object in the virtual space by changing theposition or the orientation of the tracking sensor 36 a.

In one embodiment, the video generation unit 21 d constructs a virtualspace based on tracking information of the tracking sensor 36 a. Forexample, the video generation unit 21 d determines configurationinformation (the position in the virtual space, a gaze position, agazing direction, and the angle of view) of the virtual camera based onthe tracking information of the tracking sensor 36 a. Moreover, thevideo generation unit 21 d determines a rendering area in the entirevirtual space based on the configuration information of the virtualcamera and generates moving image information for displaying therendering area in the virtual space.

The video generation unit 21 d may be configured to determine theposition and the orientation of the first character object and thesecond character object in the virtual space, and the configurationinformation of the virtual camera based on tracking information of thetracking sensor 36 b instead of or in addition to the trackinginformation of the tracking sensor 36 a.

The video generation unit 21 d is able to include voices of the actor A1and the actor A2 collected by the microphone in the studio unit 30 withthe generated moving image.

As described above, the video generation unit 21 d generates ananimation of the first character object moving in synchronization withthe movement of the body and facial expression of the actor A1, and ananimation of the second character moving in synchronization with themovement of the body and facial expression of the actor A2. The videogeneration unit 21 d then includes the voices of the actor A1 and theactor A2 with the animations respectively to generate a video fordistribution.

The video distribution unit 21 e distributes the video generated by thevideo generation unit 21 d. The video is distributed to the clientdevices 10 a to 10 c and other client devices over the network 50. Thevideo distribution unit 21 e refers to the list of users who haverequested distribution of the video (a distribution destination list),and distributes the video to the client devices of the users included inthe list. As will be described later, there may be a case wheredistribution of the video to a specific user is prohibited by aninstruction from the supporter computer 40. In this case, the video isdistributed to users other than the user to whom distribution of thevideo is prohibited. The received video is reproduced in the clientdevices 10 a to 10 c.

The video may be distributed to a client device (not shown) installed inthe studio room R, and projected from the client device onto the screenS via a short focus projector. The video may also be distributed to thesupporter computer 40. In this way, the supporter B1 and the supporterB2 can check the viewing screen of the distributed video.

An example of the screen on which the video distributed from the serverdevice 20 to the client device 10 a and reproduced by the client device10 a is displayed is illustrated in FIG. 5 . As shown in FIG. 7 , adisplay image 70 of the video distributed from the server device 20 isdisplayed on the display of the client device 10 a. The display image 70displayed on the client device 10 a includes a character object 71Acorresponding to the actor A1, a character object 71B corresponding tothe actor A2, a table object 72 a representing a table, in a virtualspace. The object 72 is not a gift object, but is one of objects usedfor constructing a virtual space included in the object data 23 b. Thecharacter object 71A is generated by applying the first body motion dataand the first face motion data of the actor A1 to the model data for theactor A1 included in the model data 23 a. The character object 71A ismotion controlled based on the first body motion data and the first facemotion data. The character object 71B is generated by applying thesecond body motion data and the second face motion data of the actor A2to the model data for the actor A2 included in the model data 23 a. Thecharacter object 71B is motion controlled based on the second bodymotion data and the second face motion data. Thus, the character object71A is controlled to move in the screen in synchronization with themotions of the body and facial expression of the actor A1, and thecharacter object 71B is controlled to move in the screen insynchronization with the motions of the body and facial expression ofthe actor A2.

As described above, the video from the server device 20 may bedistributed to the supporter computer 40. The video distributed to thesupporter computer 40 is displayed on the supporter computer 40 in thesame manner as FIG. 5 . The supporter B1 and the supporter B2 are ableto change the configurations of the components of the studio unit 30while viewing the video reproduced by the supporter computer 40. In oneembodiment, when the supporter B1 and the supporter B2 wish to changethe angle of the character object 71A and the character object 71B inthe live distributed video, they can cause an instruction signal tochange the orientation of the tracking sensor 36 a to be sent from thesupporter computer 40 to the tracking sensor 36 a. The tracking sensor36 a is able to change its orientation in accordance with theinstruction signal. For example, the tracking sensor 36 a may berotatably attached to a stand via a pivoting mechanism that includes anactuator disposed around the axis of the stand. When the tracking sensor36 a received an instruction signal instructing to change itsorientation, the actuator of the pivoting mechanism may be driven basedon the instruction signal, and the tracking sensor 36 a may be turned byan angle according to the instruction signal. In one embodiment, thesupporter B1 and the supporter B2 may cause the supporter computer 40 totransmit an instruction for using the tracking information of thetracking sensor 36 b to the tracking sensor 36 a and the tracking sensor36 b, instead of the tracking information from the tracking sensor 36 a.The tracking sensor 36 a and the tracking sensor 36 b may be configuredand installed so as to be movable the actor or supporter. As a result,the actor or supporter can hold and move the tracking sensor 36 a andthe tracking sensor 36 b.

In one embodiment, when the supporter B1 and the supporter B2 determinethat some instructions are needed for the actor A1 or the actor A2 asthey viewing the video reproduced on the supporter computer 40, they mayinput a message indicating the instruction(s) into the supportercomputer 40 and the message may be output to the display 39. Forexample, the supporter B1 and the supporter B2 can instruct the actor A1or the actor A2 to change his/her standing position through the messagedisplayed on the display 39.

The display request processing unit 21 f receives a display request todisplay a gift object from a client device of a viewing user, andperforms processing according to the display request. Each viewing useris able to transmit a display request to display a gift object to theserver device 20 by operating his/her client device. For example, thefirst viewing user can transmit a display request to display a giftobject to the server device 20 by operating the client device 10 a. Thedisplay request to display a gift object may include the user ID of theviewing user and the identification information (object ID) thatidentifies the object for which the display request is made.

As described above, the gift object may include the effect object, thenormal object, and the decorative object. The effect object and thenormal object are examples of the first object. In addition, a displayrequest for requesting display of the effect object or the normal objectis an example of a second display request.

In one embodiment, when the display request processing unit 21 freceived a display request to display a specific effect object from aviewing user, the display request processing unit 21 f performs aprocess, in response to the display request, to display the effectobject for which the display request is made in the display image 70 ofthe video. For example, when a display request to display an effectobject simulating confetti is made, the display request processing unit21 f displays in the display image 70 an effect object 73 simulatingconfetti based on the display request as shown in FIG. 6 .

In one embodiment, when the display request processing unit 21 freceived a display request to display a specific normal object from aviewing user, the display request processing unit 21 f performs aprocess, in response to the display request, to display the normalobject for which the display request is made in the video 70. Forexample, when a display request to display a normal object simulating astuffed bear is made, the display request processing unit 21 f displaysa normal object 74 simulating a stuffed bear in the display image 70based on the display request as shown in FIG. 6 .

The display request for the normal object 74 may include a displayposition specifying parameter for specifying the display position of thenormal object 74 in the virtual space. In this case, the display requestprocessing unit 21 f displays the normal object 74 at the positionspecified by the display position specifying parameter in the virtualspace. For example, the display position specifying parameter mayspecify the upper position of the table object 72 a representing a tableas the display position of the normal object 74. A viewing user is ableto specify the position where the normal object is to be displayed byusing the display position specifying parameter while watching thelayouts of the character object 71A, the character object 71B, the giftobject, and other objects included in the video 70.

In one embodiment, the normal object 74 may be displayed such that itmoves within the display image 70 of the video. For example, the normalobject 74 may be displayed such that it falls from the top to the bottomof the screen. In this case, the normal object 74 may be displayed inthe display image 70 during the fall, which is from when the objectstarts to fall and to when the object has fallen to the floor of thevirtual space of the video 70, and may disappear from the display image70 after it has fallen to the floor. A viewing user can view the fallingnormal object 74 from the start of the fall to the end of the fall. Themoving direction of the normal object 74 in the screen can be specifiedas desired. For example, the normal object 74 may be displayed in thedisplay image 70 so as to move from the left to the right, the right tothe left, the upper left to the lower left, or any other direction ofthe video 70. The normal object 74 may move on various paths. Forexample, the normal object 74 can move on a linear path, a circularpath, an elliptical path, a spiral path, or any other paths. The viewinguser may include, in the display request to display the normal object, amoving direction parameter that specifies the moving direction of thenormal object 74 and/or a path parameter that specifies the path onwhich the normal object 74 moves, in addition to or in place of thedisplay position specifying parameter. In one embodiment, among theeffect objects and the normal objects, those whose size in the virtualspace is smaller than a reference size (for example, a piece of paper ofconfetti of the effect object 73) may be displayed such that a part orall of the object(s) is overlapped with the character object 71A and/orthe character object 71B. In one embodiment, among the effect objectsand the normal objects, those whose size in the virtual space is largerthan the reference size (for example, the normal object 74 (the stuffedbear)) may be displayed at a position where the object is not overlappedwith the character object. In one embodiment, among the effect objectsand the normal objects, if those whose size in the virtual space islarger than the reference size (for example, the normal object 74 (thestuffed bear)) is overlapped with the character object 71A and/or thecharacter object 71B, the object is displayed behind the overlappingcharacter object.

In one embodiment, when the display request processing unit 21 freceived a display request to display a specific decorative object froma viewing user, the display request processing unit 21 f adds thedecorative object for which the display request is made to the candidatelist 23 d based on the display request. The display request to displaythe decorative object is an example of a first display request. Forexample, the display request processing unit 21 f may storeidentification information (object ID) identifying the specificdecorative object for which the display request has been made from theviewing user in the candidate list 23 d in association with the user IDof the viewing user (see FIG. 4 ). When more than one display request todisplay a decorative object is made, for each of the display requests,the user ID of the viewing user who made the display request and thedecorative object ID of the decoration object for which the displayrequest is made by the viewing user are associated with each other andstored in the candidate list 23 d.

In one embodiment, in response to one or more of the decorative objectsincluded in the candidate list 23 d being selected, the decorativeobject selection unit 21 g performs a process to display the selecteddecorative object in the display image 70 of the video. In thespecification, a decorative object selected from the candidate list 23 dmay be referred to as a “selected decorative object.”

The selection of the decoration object from the candidate list 23 d ismade, for example, by the supporter B1 and/or the supporter B2 whooperate the supporter computer 40. In one embodiment, the supportercomputer 40 displays a decorative object selection screen. FIG. 8 showsan example of a decorative object selection screen 80 in one embodiment.The decorative object selection screen 80 is displayed, for example, onthe display of the supporter computer 40. The decorative objectselection screen 80 shows, for example, each of the plurality ofdecoration objects included in the candidate list 23 d in a tabularform. As illustrated, the decorative object selection screen 80 in oneembodiment includes a first column 81 showing the type of the decorationobject, a second column 82 showing the image of the decoration object,and a third column 83 showing the body part of a character objectassociated with the decoration object. Further, on the decorative objectselection screen 80, selection buttons 84 a to 84 c for selecting eachdecoration object are displayed. Thus, the decorative object selectionscreen 80 displays decorative objects that can be selected as theselected decorative object.

The supporters B1 and B2 are able to select one or more of thedecorative objects shown on the decoration object selection screen 80.For example, the supporter B1 and the supporter B2 are able to select aheadband by selecting the selection button 84 a. When it is detected bythe decorative object selection unit 21 g that the headband is selected,the display request processing unit 21 f displays the selecteddecorative object 75 that simulates the selected headband on the displayscreen 70 of the video, as shown in FIG. 7 . The selected decorativeobject 75 is displayed on the display image 70 in association with aspecific body part of a character object. The selected decorative object75 may be displayed such that it contacts with the specific body part ofthe character object. For example, since the selected decorative object75 simulating the headband is associated with the head of the characterobject, it is attached to the head of the character object 71A as shownin FIG. 7 . The decorative object may be displayed on the display screen70 such that it moves along with the motion of the specific part of thecharacter object. For example, when the head of the character object 71Awith the headband moves, the selected decorative object 75 simulatingthe headband moves in accordance with the motion of the head of thecharacter object 71A as if the headband is attached to the head of thecharacter object 71A.

The selected decorative object 75 may be displayed on the display screen70 in association with the character object 71B instead of the characterobject 71A. Alternatively, the selected decorative object 75 may bedisplayed on the display screen 70 in association with the characterobject 71A and the character object 71B.

In one embodiment, the decorative object selection screen 80 may beconfigured to exclude information identifying a user who holds thedecorative object or a user who has made a display request to displaythe decorative object. By configuring the decorative object selectionscreen 80 in this manner, it is possible to prevent a selector fromgiving preference to a particular user when selecting a decorativeobject.

In one embodiment, the decorative object selection screen 80 maydisplay, for each decorative object, information regarding a user whoholds the decorative object or a user who made a display request for thedecorative object. Such information displayed for each decorative objectmay include, for example, the number of times that the user who madethis display request of the decorative object has made display requestsof the decorative object so far and the number of times that thedecorative object has been actually selected (for example, informationindicating that the display request to display the decorative object hasbeen made five times and the decorative object has been selected twotimes among the five times), the number of times that the user views thevideo of the character object 71A and/or the character object 71B, thenumber of times that the user views a video (regardless of whether thecharacter object 71A and/or the character object 71B appears in thevideo or not), the amount of money which the user spent for the giftobject, the number of times that the user purchases the object, thepoints possessed by the user that can be used in the video distributionsystem 1, the level of the user in the video distribution system 1, andany other information about the user who made the display request todisplay the respective decorative object may be displayed. According tothis embodiment, it is possible to select the decorative object based onthe behavior and/or the viewing history of the user who has made thedisplay request of the decorative object in the video distributionsystem 1.

In one embodiment, a constraint(s) may be imposed on the display ofdecorative objects to eliminate overlapping. For example, with regard tothe character object 71A, if a decorative object associated with thespecific body part of the character object is already selected,selection of other decorative objects associated with the specific bodypart may be prohibited. As shown in the embodiment of FIG. 7 , when theheadband associated with the “head” of the character object 71B isalready selected, the other decorative objects associated with the“head” (for example, a decorative object simulating a “hat” associatedwith the head) are not displayed on the decorative object selectionscreen 80, or a selection button for selecting the decorative objectsimulating the hat is made unselectable on decorative object selectionscreen 80. According to this embodiment, it is possible to prevent thedecorative object from being displayed so as to overlap with a specificpart of the character object.

The decorative object selection screen 80 may be displayed on anotherdevice instead of or in addition to the supporter computer 40. Forexample, the decorative object selection screen 80 may be displayed onthe display 39 and/or the screen S in the studio room R. In this case,the actor A1 and the actor A2 are able to select a desired decorativeobject based on the decorative object selection screen 80 displayed onthe display 39 or the screen S. Selection of the decorative object bythe actor A1 and the actor A2 maybe made, for example, by operating thecontroller 33 a, the controller 33 b, the controller 34 a, or thecontroller 34 b.

In one embodiment, in response to a request from a viewing user of thevideo, the object purchase processing unit 21 h transmits, to a clientdevice of the viewing user (for example, the client device 10 a),purchase information of each of the plurality of gift objects that canbe purchased in relation to the video. The purchase information of eachgift object may include the type of the gift object (the effect object,the normal object, or the decorative object), the image of the giftobject, the price of the gift object, and any other informationnecessary to purchase the gift object. The viewing user is able toselect a gift object to purchase it considering the gift object purchaseinformation displayed on the client device 10 a. The selection of thegift objects which the viewing user purchases may be performed byoperating the client device 10 a. When a gift object to be purchased isselected by the viewing user, a purchase request for the gift object istransmitted to the server device 20. The object purchase processing unit21 h performs a payment process based on the purchase request. When thepayment process is completed, the purchased gift object is held by theviewing user. In this case, the object ID of the purchased gift objectis stored in the possession list 23 c in association with the user ID ofthe viewing user who purchased the object.

Gift objects that can be purchased may be different for each video. Thegift objects may be made purchasable in two or more different videos.That is, the purchasable gift objects may include a gift object uniqueto each video and a common gift object that can be purchased in thevideos. For example, the effect object that simulates confetti may bethe common gift object that can be purchased in the two or moredifferent videos.

In one embodiment, when a user purchases an effect object while viewinga video, the purchased effect object may be displayed automatically inthe video that the user is viewing in response to completion of thepayment process to purchase the effect object. In the same manner, whena user purchases a normal object while viewing a video, the purchasednormal object may be automatically displayed in the video that the useris viewing in response to completion of the payment process to purchasethe normal object.

In another embodiment, in response to completion of the payment processperformed by the object purchase processing unit 21 h for the effectobject to be purchased, a notification of the completion of the paymentprocess may be sent to the client device 10 a, and a confirmation screenmay be displayed to confirm whether the viewing user wants to make adisplay request to display the purchased effect object on the clientdevice 10 a. When the viewing user selects to make the display requestfor the purchased effect object, the display request to display thepurchased effect object may be sent from the client device of theviewing user to the display request processing unit 21 f, and thedisplay request processing unit 21 f may perform the process to displaythe purchased effect object in the display image 70 of the video. Evenwhen the normal object is to be purchased, a confirmation screen may bedisplayed on the client device 10 a to confirm whether the viewing userwants to make a display request to display the purchased normal object,in the same manner as above.

Next, the supporter computer 40 in one embodiment will be described. Inone embodiment, the supporter computer 40 includes a computer processor41, a communication I/F 42, a storage 43, a display 44, and an inputinterface 45.

Similarly to the computer processor 21, the computer processor 41 may beany computing device such as a CPU. Similarly to the communication I/F22, the communication I/F 42 may be, a driver, software, or acombination thereof for communicating with other devices. Similarly tothe storage 23, the storage 43 may be a storage device capable ofstoring data such as a magnetic disk. The display 44 may be a liquidcrystal display, an organic EL display, an inorganic EL display or anyother display device capable of displaying images. The input interface45 may be any input interface that receives input from the supportersuch as a mouse and a keyboard. The display 44 is an example of a seconddisplay device.

FIG. 9 illustrates an example of an image displayed on the display 44 ofthe supporter computer 40. In the illustrated embodiment, a displayimage 46 displayed on the display 44 includes a first display area 47 afor displaying a display screen of a video in a first type client device(for example, a personal computer), a second display area 47 b fordisplaying a display screen of the video in a second type client device(for example, a smart phone), and a plurality of operation icons 48.

By monitoring the video displayed in the first display area 47 a, thesupporter is able to check whether the video distributed to the firsttype client device is normally displayed. Similarly, by monitoring thevideo displayed in the second display area 47 b, the supporter is ableto check whether the video distributed to the second type client deviceis normally displayed.

In the illustrated embodiment, the operation icon 48 includes a firstoperation icon 48 a, a second operation icon 48 b, a third operationicon 48 c, a fourth operation icon 48 d, and a fifth operation icon 48e. The supporter is able to select a desired operation icon via theinput interface 45. When the first operation icon 48 a, the secondoperation icon 48 b, the third operation icon 48 c, the fourth operationicon 48 d, and the fifth operation icon 48 e are selected by thesupporter, the supporter computer 40 receives a first operation input, asecond operation input, a third operation input, a fourth operationinput, and a fifth operation input, respectively.

Functions realized by the computer processor 41 will be now describedmore specifically. The computer processor 41 functions as an additionalinformation display unit 41 a and a distribution management unit 41 b byexecuting computer-readable instructions included in a predeterminedprogram. At least some of the functions that can be realized by thecomputer processor 41 may be realized by a computer processor other thanthe computer processor 41 of the video distribution system 1. At leastsome of the functions realized by the computer processor 41 may berealized by, for example, the computer processor 21.

The additional information display unit 41 a is configured to addvarious additional information to the video in accordance with variousoperation inputs from the supporter via the input interface 45.

In one embodiment, when the supporter computer 40 receives the firstoperation input made by selecting the first operation icon 48 a, theadditional information display unit 41 a is configured to display amessage 101 in the video displayed on the display 39 based on the firstoperation input, while the additional information display unit 41 amakes the message 101 undisplayed on the client device. The message 101is an example of first additional information.

FIG. 10 shows an example of a display image of the video displayed onthe display 39. The display image 100 shown in FIG. 10 includes themessage 101. The display image 100 is similar to the display image 70(see FIG. 5 ) of the video distributed and displayed on the clientdevice of the viewing user at the same time except that the displayimage 100 includes the message 101. That is, the display image 100displayed on the display 39 is configured by adding the message 101 tothe video being distributed.

The message 101 is, for example, a text message input by a supporter.The message 101 may include image information instead of text formatinformation or in addition to the text format information. The message101 may include an instruction on an actor's performance, an instructionon an actor's statement, an instruction on an actor's position, andvarious other instructions or messages related to the live distributedvideo.

In one embodiment, when a supporter selects the first operation icon 48a on the supporter computer 40, a window that allows the supporter toinput a message is displayed on the display 44 of the supporter computer40. The supporter is able to input a message in a message input area ofthe window by operating the input interface 45. The inputted message istransmitted from the supporter computer 40 to the display 39. Thedisplay 39 and the client devices 10 a to 10 c display the videodistributed by the video distribution unit 21 e. On the display 39, themessage received from the supporter computer 40 is displayed as amessage 101 in a predetermined area of the video distributed by thevideo distribution unit 21 e as shown in FIG. 10 . Whereas in the videodisplayed on the client devices 10 a to 10 c, the message 101 orinformation corresponding thereto is not displayed. According to thisembodiment, it is possible to communicate instructions and messagesregarding the live distributed video to the actor or other distributionstaff member who can see the display 39 without affecting the videoviewed by viewing users of the client devices.

Subsequently, display of the second additional information will bedescribed with reference to FIGS. 11 a and 11 b . In one embodiment,once the second operation icon 48 b is selected and the second operationinput is received by the supporter computer 40, the additionalinformation display unit 41 a is configured to display an interruptionimage in the video displayed on the display 39 and the video displayedon the client devices based on the second operation input. Theinterruption image is an example of the second additional information.The second operation input may be input to the server device 20 or thesupporter computer 40 in accordance with operation of the controller 33a or 33 b by the actor A1 or operation of the controller 34 a or 34 b bythe actor A2.

FIG. 11 a shows an example of a display image 110 a displayed on theclient device 10 a when the second operation input is received, and FIG.11 b shows an example of a display image 110 b displayed on the display39 when the second operation input received. Both the display image 110a and the display image 110 b include an interruption image 111. Theinterruption image 111 is disposed in the top layer of the livedistributed video. Thus, the interruption image 111 is displayed on thescreen of the client device and the display 39 on which the video isbeing played. Although there may be a slight visual difference betweenthe interruption image 111 displayed on the client device and theinterruption image 111 displayed on the display 39 due to a differencein the performance of the display device, both are substantially thesame image.

In the illustrated embodiment, the interruption image 111 is an imagethat is displayed in emergency instead of displaying a video containinga normal virtual space and a character object when an unexpectedsituation occurs during a live distribution of the video. The supporteris able to select the second operation icon 48 b on the supportercomputer 40, for example, when trouble occurs in motion control of thecharacter object due to a failure of equipment used in the studio roomR. The additional information display unit 41 a performs processing fordisplaying the interruption image 111 on each client device and thedisplay 39 in response to the selection of the second operation icon 48b. In response to reception of the second operation input, theadditional information display unit 41 a may cause the interruptionimage 111 to be superimposed on the live-distributed video. In responseto reception of the second operation input, the additional informationdisplay unit 41 a may transmit a control signal for displaying theinterruption image 111 to the client device. Upon receipt of the controlsignal, the client device that received the control signal may perform aprocess to superimpose the interruption image 111 on the video beingplayed or a process to display the interruption image 111 instead of thevideo being played.

When the first operation input is accepted by the selection of the firstoperation icon 48 a while the interrupt image 111 is displayed on thedisplay 39, the additional information display unit 41 a causes thedisplay 39 to superimpose on the interrupt image 111 to be a message.101 may be displayed.

In the above embodiment, when an unexpected situation occurs during alive distribution of a video, the interruption image 111 that isdisplayed in emergency is distributed to the client devices and thedisplay 39 instead of continuing to distribute the video. Thedistributor can handle the situation while the interruption image 111 isdisplayed.

Display of the second additional information will be now described withreference to FIGS. 11 a and 11 b . In one embodiment, when the thirdoperation icon 48 c is selected and the third operation input isreceived by the supporter computer 40, the additional informationdisplay unit 41 a is configured to generate a modified character objectand performs processing for distribution of a video that contains themodified character object. The third operation input may be input to theserver device 20 or the supporter computer 40 in accordance withoperation of the controller 33 a or 33 b by the actor A1 or operation ofthe controller 34 a or 34 b by the actor A2.

FIG. 12 a shows another example of a display image 120 a displayed onthe client device 10 a, and FIG. 12 b shows an example of a displayimage 120 b displayed on the client device 10 a. The display image 120 aof FIG. 12 a includes the character object 71B, and the display image120 a of FIG. 12 b includes a modified character object 171B which willbe described later. As described above, facial expression of thecharacter object 71B is controlled so as to change in synchronizationwith the change of the facial expression of the actor A2 based on theface motion data of the actor A2. In this specification, all or part ofthe portion of the character object that changes based on the facemotion data may be referred to as a “face portion.” In the illustratedembodiment, motion control is performed such that eyes of the characterobject 71B moves in synchronization with the eye motion of the actor A2,so that the face portion 121 is set in the position where includes botheyes of the character object 71B. The face portion 121 may be set toinclude the entire face of the character object 71B.

In the display image 120 a shown in FIG. 12 a , the image of the eyes ofthe character object 71B displayed in the face portion 121 is generatedby applying the face motion data to the model data. Whereas the image ofthe eyes displayed in the face portion 121 in the display image 120 b ofFIG. 12 b is a prepared image 122 b that is prepared in advance beforethe start of the video distribution to be fitted in the face portion121. The prepared image 122 b may be stored in the storage 23. Theprepared image 122 b is an example of third additional information.

In one embodiment, the additional information display unit 41 a isconfigured to display the prepared image 122 b composited in the faceportion 121 of the character object 71B when the third operation inputis received. For example, when the third operation input is received,the additional information display unit 41 a transmits a modificationinstruction to the animation generation unit 21 c, and the animationgeneration unit 21 c composites the prepared image 122 b to be fitted inthe face portion 121 of the character object 71B in accordance with themodification instruction from the additional information display unit 41a instead of using the image generated based on the face motion data inorder to generate an animation of the modified character object 171B. Inthis specification, a character object into which the prepared image 122b is inserted, not the image generated based on the face motion data,may be referred to as a modified character object. The modifiedcharacter object is an object in which a part of the character objectgenerated by the animation generation unit 21 c is modified. In theabove example, the animation of the modified character object 171B isgenerated by compositing the prepared image 122 b to be displayed in theface portion 121 of the character object 71B generated by the animationgeneration unit 21 c. In response to reception of the third operationinput, the additional information display unit 41 a may transmit acontrol signal for displaying the prepared image 122 b to the clientdevice. Upon receipt of the control signal, the client device mayperform a process to composite the prepared image 122 b to be displayedin the character object in the video being played.

The process of applying the face motion data to the model data to changethe facial expression of a character object imposes a high processingload on the processor. For this reason, the facial expression of thecharacter object may fail to follow the facial expression of the actorin a timely manner. Since character's voice and motions other than thefacial expression can follow timely to the voice and body motions of theactor movement, if the facial expression of the character object failsto follow the facial expression of the actor in a timely manner, it maygive the viewing users a feeling of strangeness. In the aboveembodiment, when the facial expression of the character object fails tofollow the facial expression of the actor timely, the modified characterobject that incorporates the prepared image 122 b to be fitted thereinis generated, and the video containing the modified character object isdistributed. In this way, it is possible to prevent deterioration of thequality of the video caused by the fact that the facial expression ofthe character object fails to follow the facial expression of the actor.

The storage 23 may store a plurality of different types of images ascandidates of the prepared image 122 b to be composited. The candidatesof the prepared image 122 b are displayed on the display 44 of thesupporter computer 40, and an image selected by the supporter from amongthe candidates may be used as the prepared image 122 b to be composited.

Display of a blind object will be now described with reference to FIG.13 . In one embodiment, when the fourth operation icon 48 c is selectedand the fourth operation input is received by the additional informationdisplay unit 41 a, the additional information display unit 41 a displaysa blind object that is used to hide at least a part of the characterobject that wears the decorative object. The fourth operation input maybe input to the server device 20 or the supporter computer 40 inaccordance with operation of the controller 33 a or 33 b by the actor A1or operation of the controller 34 a or 34 b by the actor A2. In responseto reception of the fourth operation input, the additional informationdisplay unit 41 a may transmit a control signal for displaying theprepared image 122 b to the client device. Upon receipt of the controlsignal, the client device that received the control signal may displaythe blind object such that the blind object is superimposed on the videobeing played.

In the embodiment of FIG. 13 , it is assumed that a T-shirt is selectedas the decoration object to be attached to the character object 71B inthe selection screen 80. The additional information display unit 41 amay display the blind object such that the blind object hides a part ofthe character object 71B that is associated with the selected decorativeobject. In the display image 130 of FIG. 13 , a blind object 131 isdisplayed such that at least the torso of the character object 71Bassociated with the T-shirt, which is the selected decorative object, ishidden by the blind object 131.

A process for prohibiting the distribution of a video to a banned userwill be described. In one embodiment, when the fifth operation icon 48 eis selected and a fifth operation input is received by the distributionmanagement unit 41 b, the distribution management unit 41 b performs aprocess for prohibiting distribution of the video to a specific viewinguser or a specific client device. In one embodiment, when the fifthoperation icon 48 e is selected during distribution of a video, a listof users who are viewing the video is displayed. The distributionmanagement unit 41 b performs a process for ceasing the distribution ofthe video to a user(s) selected by the supporter or the actor from thelist. For example, the distribution management unit 41 b may flag theselected user (distribution banned user) to identify the user in thedistribution destination list of the video. The video distribution unit21 e may distribute the video to users with no flag that is used toidentify the banned user, among the users included in the distributiondestination list of the video. In this way, the distribution of thevideo to the banned user is stopped. In another embodiment, thedistribution management unit 41 b may be configured to make a user(s)selected by the supporter or the actor from the list of the users whoare viewing the live distributed video inaccessible to some of thefunctions that are normally available to the users. For example, thedistribution management unit 41 b may be configured to prohibit a userwho has posted an inappropriate message from posting of a new message.The user who is prohibited from posting a message is allowed to continueviewing the video even after the prohibition, but is no longer allowedto post a message on the video.

Next, with reference to FIGS. 14 to 19 , a video distribution process inone embodiment will be described. FIG. 14 is a flow chart showing a flowof a video distribution process in one embodiment, FIG. 15 is aflowchart of a process of displaying the first additional information inone embodiment, FIG. 16 is a flowchart of a process of displaying thesecond additional information in one embodiment, FIG. 17 is a flowchartof a process of displaying the third additional information in oneembodiment, FIG. 18 is a flowchart of a process of displaying the blindobject in one embodiment, and FIG. 19 is a flow chart showing a flow ofa process for prohibiting the video distribution to a banned user in oneembodiment.

First, a video distribution process in one embodiment will be describedwith reference to FIG. 14 , In the video distribution process, it isassumed that the actor A1 and the actor A2 are giving performances inthe studio room R. First, in step S11, body motion data, which is adigital representation of the body motions of the actor A1 and the actorA2, and face motion data, which is a digital representation of thefacial motions (expression) of the actor A1 and the actor A2, aregenerated. Generation of the body motion data is performed, for example,by the body motion data generation unit 21 a described above, andgeneration of the face motion data is performed, for example, by theface motion data generation unit 21 b described above.

Next, in step S12, the body motion data and the face motion data of theactor A1 are applied to the model data for the actor A1 to generateanimation of the first character object that moves in synchronizationwith the motions of the body and facial expression of the actor A1.Similarly, the body motion data and the face motion data of the actor A2are applied to the model data for the actor A2 to generate animation ofthe second character object that moves in synchronization with themotions of the body and facial expression of the actor A2. Thegeneration of the animation is performed, for example, by theabove-described animation generation unit 21 c.

Next, in step S13, a video including the animation of the firstcharacter object corresponding to the actor A1 and the animation of thesecond character object corresponding to the actor A2 is generated. Thevoices of the actor A1 and the actor A2 may be included in the video.The animation of the first character object and the animation of thesecond character object may be provided in the virtual space. Generationof the video is performed, for example, by the above-described videogeneration unit 21 d.

Next, the process proceeds to step S14 and the video generated in stepS13 is distributed. The video is distributed to the client devices 10 ato 10 c and other client devices over the network 50. The video may bedistributed to the supporter computer 40 and/or may be projected on thescreen S in the studio room R. The video is distributed continuouslyover a predetermined distribution period. The distribution period of thevideo may be set to, for example, 30 seconds, 1 minute, 5 minutes, 10minutes, 30 minutes, 60 minutes, 120 minutes, and any other length oftime.

Subsequently in step S15, it is determined whether a terminationcondition for ending the distribution of the video is satisfied. The endcondition is, for example, that the distribution ending time has come,that the supporter computer 40 has issued an instruction to end thedistribution, or any other conditions. If the end condition is notsatisfied, the steps S11 to S14 of the process are repeatedly executed,and distribution of the video including the animation synchronized withthe motions of the actor A1 and the actor A2 is continued. When it isdetermined that the end condition is satisfied for the video, thedistribution process of the video is ended.

Next, with reference to FIG. 15 , a description is given of a process ofdisplaying the first additional information that is performed during avideo live-distribution. In step S21, it is determined whether the firstoperation input has been made during the video live-distribution. Forexample, when the first operation icon 48 a is selected on the supportercomputer 40, it is determined that the first operation input has beenmade. When the first operation input has been made, the display processof the first additional information proceeds to step S22. In step S22,the message 101, which is an example of the first additionalinformation, is shown on the video displayed on the display 39, and thedisplay process to display the first additional information ends. Themessage 101 is not displayed on the client device at this time.

With reference to FIG. 16 , a process of displaying the secondadditional information during a video live-distribution will be nowdescribed. In step S31, it is determined whether the second operationinput has been made during the video live-distribution. For example,when the second operation icon 48 b is selected on the supportercomputer 40, it is determined that the second operation input has beenmade. When the second operation input has been made, the display processof the second additional information proceeds to step S32. In step S32,the interruption image 111, which is an example of the second additionalinformation, is displayed on the client device(s) and the display 39,and the display process to display the second additional informationends.

With reference to FIG. 17 , a process of displaying the third additionalinformation during a video live-distribution will be now described. Instep S41, it is determined whether the third operation input has beenmade during the video live-distribution. For example, when the thirdoperation icon 48 c is selected on the supporter computer 40, it isdetermined that the third operation input has been made. When the thirdoperation input has been made, the display process to display the thirdadditional information proceeds to step S42. In step S42, the preparedimage 122 b, which is an example of the third additional information, isdisplayed in the face portion of the character object, and the displayprocess to display the third additional information ends.

Next, with reference to FIG. 18 , a description is given of a displayprocess to display the blind object during a video live-distribution. Instep S51, it is determined whether selection of a decorative object thatis to be attached to the character object appearing in the video hasbeen made during the video live-distribution. As described above, thedecorative object attached to the character object may be selected fromamong the candidates in the candidate list 23 d. The process to selectthe decorative object is performed, for example, by the above-mentioneddecorative object selection unit 21 g.

When it is determined in step S51 that the decorative object has beenselected, the process proceeds to step S52. In step S52, it isdetermined whether the fourth operation input has been made. Forexample, when the fourth operation icon 48 d is selected on thesupporter computer 40, it is determined that the fourth operation inputhas been made. When the fourth operation input has been made, thedisplay process to display the blind object proceeds to step S53. Inanother embodiment, when the decorative object is selected from thecandidate list 23 d, it may be determined that the fourth operationinput has been made. In this case, step S52 is omitted, and the displayprocess to display the blind object proceeds from step S51 to step S53.

In step S53, the blind object 131 is displayed such that it hides atleast a part of the character object. For example, when the decorativeobject selected in step S51 is a T-shirt, the blind object 131 is addedto the video so as to hide the torso of the character object where isassociated with the T-shirt.

With reference to FIG. 19 , the process for prohibiting the distributionof a video to a specific user will be further described. In step S61, itis determined whether the fifth operation input has been made during avideo live-distribution. For example, when the fifth operation icon 48 eis selected on the supporter computer 40, it is determined that thefifth operation input has been made.

When the fifth operation input has been made, the distributionprohibition process proceeds to step S62. In step S62, a user (banneduser) to whom the video is prohibited from being distributed isdesignated. For example, a list of users who are watching the video isdisplayed on the display 44 of the supporter computer 40, and a userselected by the supporter from this list is designated as the banneduser. The process for designating the banned user is performed by, forexample, the distribution management unit 41 b.

Next, the distribution prohibition process proceeds to step S63. In stepS63, the video is distributed only to users who are not designated asthe banned users among the users included in the distributiondestination list of the video. The video distribution process isperformed, for example, by the above-described video distribution unit21 e.

With the video distribution system 1 in the above embodiment, it is easyto handle with an unexpected situation occurred during a livedistribution. For example, by displaying the message 101 on the display39 without displaying the message 101 on the client device(s), it ispossible to communicate instructions and messages regarding the livedistributed video to the actor or other distribution staff member whocan see the display 39 without affecting the video viewed by viewingusers.

In the above-described another embodiment, when an unexpected situationoccurs during a live distribution of a video, the interruption image 111displayed in emergency is distributed to the client devices and thedisplay 39 instead of continuing to distribute the video.

In the above-described yet another embodiment, when the facialexpression of the character object fails to follow the facial expressionof the actor timely, the modified character object that incorporates theprepared image 122 b to be fitted therein is generated, and the videocontaining the modified character object is distributed. In this way, itis possible to prevent deterioration of the quality of the video causedby the fact that the facial expression of the character object fails tofollow the facial expression of the actor.

In the above-described still yet another embodiment, even when someviewing users take inappropriate actions such as posting inappropriatemessages, it is possible to stop the distribution of the video to suchviewing users.

Embodiments of the invention are not limited to the above-describedembodiments, and various modifications can be made without departingfrom the scope of the invention. For example, shooting and production ofthe video to be distributed may be performed outside the studio room R.For example, video shooting to generate a video to be distributed may beperformed at an actor's or supporter's home.

The procedures described herein, particularly those described with aflowchart, are susceptible of omission of part of the steps constitutingthe procedure, adding steps not explicitly included in the stepsconstituting the procedure, and/or reordering the steps. The proceduresubjected to such omission, addition, or reordering is also included inthe scope of the present invention unless diverged from the purport ofthe present invention.

What is claimed is:
 1. A video distribution system, comprising: adistribution server configured to cause a first display device and aplurality of client devices to play a live video, the live videocontaining an animation of a character object generated based on one ormore motions of an actor, the plurality of client devices includes afirst client device used by a first user; a storage storing one or moredecorative objects, each of the one or more decorative objects capableof being displayed in association with the character object; a supportercomputer comprising a second display device, the supporter computerbeing located remotely from the first display device, the supportercomputer being configured to (i) display the live video on the seconddisplay, (ii) cause first additional information to be displayed in thelive video that is being displayed to the actor on the first displaydevice, (iii) cause the first additional information to not be displayedon any of the plurality of client devices; (iv) allow a supporter userto input a first operation input; and (v) display in the live video,upon selection of one of the one or more decorative objects and receiptof the first operation input, a blind object for hiding at least a partof the character object.
 2. The video distribution system of claim 1,wherein the supporter computer is configured to cause second additionalinformation to be displayed (i) in the live video that is beingdisplayed on the first display device, and (ii) in the live video thatis being displayed on the first client device based on a secondoperation input.
 3. The video distribution system of claim 1, whereinthe supporter computer is configured to cause a modified characterobject formed by modifying at least a part of the character object to begenerated based on a third operation input, and cause the modifiedcharacter object to be included in the live video.
 4. The videodistribution system of claim 3, wherein the character object has a faceportion that is generated so as to move based on face motion datarepresenting face motions of the actor, and wherein the distributionserver is configured to cause third additional information to bedisplayed in the face portion based on the third operation input.
 5. Thevideo distribution system of claim 1, wherein: the plurality of clientdevices includes a second client device used by a second user; thedistribution server is configured to cause the second client device toplay the live video; the first client device is a first type device; thesecond client device is a second type device different from the firsttype device; and the supporter computer includes a second display devicethat is configured to cause a display image to be displayed, wherein thedisplay image comprises the live video displayed on the first clientdevice and a display image of the live video displayed on the secondclient device.
 6. The video distribution system of claim 1, wherein thesupporter computer prohibits the live video from being played by thefirst client device based on a fifth operation input.
 7. The videodistribution system of claim 1, wherein the first display device is usedby the actor.
 8. The video distribution system of claim 1, wherein thefirst display device displays an object that can be selected by theactor.
 9. The video distribution system of claim 1, wherein thesupporter user is not a user of any of the plurality of client devices.10. The video distribution system of claim 1, wherein the supportercomputer is operated by the supporter user.
 11. The video distributionsystem of claim 1, further comprising: one or more motion sensorsconfigured to detect the one or more motions of the actor as the one ormore motions are performed, wherein the supporter computer is furtherlocated remotely from the one or more motion sensors.
 12. The videodistribution system of claim 11, wherein the one or more motion sensorsare attached to the actor.
 13. The video distribution system of claim 1,wherein the supporter computer is located in a room proximate adifferent room where the first display device is located.
 14. The videodistribution system of claim 1, wherein: the selected one of the one ormore decorative object is associated with a specific body part of thecharacter object in the live video; and the blind object is arrangedwithin the live video so as to hide the specific part of the characterobject.
 15. A method of distributing a video performed by one or morecomputer processors executing computer readable instructions, the methodcomprising: causing a first display device and a plurality of clientdevices to play a live video, wherein the live video contains ananimation of a character object generated based on one or more motionsof an actor, and wherein the plurality of client devices includes afirst client device used by a first user; storing one or more decorativeobjects, each of the one or more decorative objects capable of beingdisplayed in association with the character object; causing the livevideo to be displayed on the first display device disposed at a positionviewable by the actor; causing the live video to be displayed on asecond display device of a supporter computer, the supporter computerbeing located remotely from the first display device; causing, by thesupporter computer, a first additional information to be displayed inthe live video that is being displayed on the first display device;causing, by the supporter computer, the first additional information tonot be displayed on any of the plurality of client devices; anddisplaying in the live video, upon selection of one of the one or moredecorative objects and receipt of an operation input by a supporter userof the supporter computer, a blind object for hiding at least a part ofthe character object.
 16. A non-transitory computer-readable storagemedium storing a distribution program causing one or more computerprocessors to: cause a first display device and a plurality of clientdevices to play a live video, wherein the live video contains ananimation of a character object generated based on one or more motionsof an actor, and wherein the plurality of client devices includes afirst client device used by a first user; store one or more decorativeobjects, each of the one or more decorative objects capable of beingdisplayed in association with the character object; cause the live videoto be displayed on the first display device disposed at a positionviewable by the actor; cause the live video to be displayed on a seconddisplay device of a supporter computer, the supporter computer beinglocated remotely from the first display device; cause, by the supportercomputer, a first additional information to be displayed in the livevideo that is being displayed on the first display device; cause, by thesupporter computer, the first additional information to not be displayedon any of the plurality of client devices; and display in the livevideo, upon selection of one of the one or more decorative objects andreceipt of an operation input by a supporter user of the supportercomputer, a blind object for hiding at least a part of the characterobject.